Rendering graphics for transfer to a display device in real-time is a complicated process that incorporates many well-developed techniques to ensure that newly generated frames are transferred from the source to the display with proper timing. Typically, the process begins with a processing unit, commonly a graphics processing unit (GPU) having a highly parallel architecture tailored to the rendering task, rendering each new frame of source content to a portion of memory known as the frame buffer. The newly generated frames of source content, referred to herein as “source frames,” are each temporarily stored in the frame buffer in sequence as images having an array of values that define the visual contents for each pixel in that particular frame. While this is occurring, these images are scanned out of the frame buffer in a process that drives the images sequentially to a display device. Meanwhile, the display device traditionally updates the image displayed on the screen periodically at a fixed frequency, known as the refresh rate, using the images that are scanned out from the frame buffer.
In order to send the rendered frames to the display, the images in the frame buffer are typically scanned out line by line and transferred serially (in sequence) over some video interface to the display device. During scanout, certain “invisible” signals are generated to govern the transfer process, so that what is actually transferred to the display device for each frame that is output from the frame buffer, referred to herein as an “output frame,” includes not only the visible pixel values of the frame's image, but other external signals which may be used by the display device to resolve how the received frame is displayed on the screen. This typically includes, among other things, a vertical synchronization signal that is pulsed between each scanned out frame image. The period of time between each scanned out frame image, i.e., between the last line or pixel of one frame image and the first line or pixel of the subsequent frame's image, is known as the “vertical blanking interval.” This vertical blanking interval is generated as part of the scanout process, and this vertical synchronization pulse used for synchronization between graphics source and display.
The frequency at which the vertical synchronization pulse occurs during scanout, and, as a result, the frequency at which the vertical blanking interval occurs, is traditionally fixed in relation to the refresh rate of the display device, so that each image scanned out from the frame buffer coincides with each refresh cycle of the display. If the frame rate of the original graphics content, i.e., the rate at which new source frames are drawn to the frame buffer by the GPU, is perfectly in sync with the refresh rate of the display, each new source frame drawn to the frame buffer by the GPU would correspond 1:1 to each image presented on the display device. For example, if the display device has a refresh rate of 60 Hz and the GPU were rendering new images to the frame buffer at a frame rate of 60 FPS in phase with the refresh cycle of the display, each image updated on the screen of the display would perfectly correspond to the source frames generated by the GPU.
However, in practice the frame rate of the source content is often variable over time and may fluctuate upward and downward, e.g., based on the complexity of the current scene or other factors associated with the generation of the frames. For example, if the current state of a video game causes too many virtual objects or too much detail within the current field of view, the frame rate may momentarily dip due to an increased computational load required to render the frame. As a result, the frame rate of the source content rendered to the frame buffer may go out of sync with the scanout of the frames from this buffer and the corresponding refresh cycles of the display device. In other words, each “source frame” that is drawn to the frame buffer may not exactly correspond to each “output frame” that is driven to the display device.
An undesirable consequence which results from this desynchronization between source frame rate and display refresh is a visual artifact known as “tearing,” aptly named because it appears as if there is a horizontal tear in the displayed image for a particular frame. Essentially, tearing occurs when a frame is scanned out of the frame buffer while that portion of memory is being updated with a new subsequent source frame, e.g., the GPU overwrites the image in the buffer with a subsequent source frame before it is finished being scanned out. As a result, the output frame that is transferred to the display device actually contains the images from two or more consecutive source frames. Correspondingly, when the display device updates its screen contents during that refresh cycle, it simultaneously contains images from different consecutive frames of the source content.
To minimize or eliminate tearing, the frame buffer commonly includes multiple buffers, i.e., a front frame buffer from which the frame images are directly scanned out, and one or more back frame buffers into which the GPU may draw new frames while a prior frame is being scanned out of the front frame buffer. When a new frame is finished rendering, a back frame buffer is swapped with the front frame buffer, e.g., by copying the contents to the front buffer or by changing a pointer value which specifies the memory address for the front buffer, so that the contents of the front buffer may be scanned out to the display device. In order to completely eliminate tearing artifacts, this is often combined with a restriction that prevents the GPU from swapping the buffers until just after a refresh cycle of the display device. This is typically accomplished by only forcing the GPU to wait for a vsync pulse occurring during the vertical blanking interval before it swaps the buffers. Since this vsync pulse and vertical blanking interval is traditionally generated at fixed intervals in relation to the refresh cycles of the display, it ensures that only whole source frames are scanned out of the frame buffer, preventing tearing artifacts from occurring.
While this is effective at preventing tearing, another problem known as “stuttering” may result, which may occur when the source frame rate drops and the scanout unit is forced to transfer an identical frame to the display. Stuttering may be especially pronounced when the GPU is restricted to only swapping the buffers between refresh cycles, since the frame rate is effectively restricted to only integral factors of the display refresh rate. Since the GPU must have a completed new source frame in order to perform the swap, if the GPU has not finished rendering the subsequent frame at the time of the synchronization pulse, it must wait another full cycle before it can swap the buffers, even if the new source frame is otherwise finished shortly thereafter. When stuttering occurs, the sudden drop in the perceived frame rate at the display can be distracting to the viewer.
In some instances, rather scanning frames out to a display device, it is desirable to send the frames to some other destination. For example, cloud gaming and other cloud-based video streaming applications may require rendered frames to be compressed and sent over a network for display in real-time, rather than transferred from the frame buffer directly to a display device. In these situations, preferably whole source frames are compressed by an encoder and sent to the remote device with minimized latency. To achieve this task, the encoder must operate on a restricted budget of resources to ensure the frames reach the remote device on time. If the source frame rate fluctuates and stuttering occurs, valuable compression resources would be wasted towards compressing an identical frame. This may result in poorer image quality in the encoded frames than might otherwise be achieved if the compression resources were more efficiently utilized. Furthermore, if identical frames are streamed over the network, limited network bandwidth is wasted on unnecessary frames.
It is within this context that aspects of the present disclosure arise.